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Abstract 


This  document  summarizes  two  days  of  discussion  between  a  number  of  cognitive  and 
behavioural  seientists  and  professional  intelligence  analysts  from  12  eountries  and  a 
multilateral  organization,  whieh  was  eo-hosted  in  Ottawa  by  DRDC,  Canada’s  Privy 
Couneil  Office,  and  the  US  Department  of  State,  February  23-24,  2009.  This 
experimental  roundtable  explored  the  question  of  whether  and  how  behavioural  and 
eognitive  seiences  ean  eontribute  to  improved  intelligenee  analysis.  It  eoneluded  that 
there  are  many  relevant  erossovers  between  these  fields,  and  the  time  and  eost  of  further 
eollaboration  and  “bridging”  between  the  seientist-intelligenee  practitioner  perspectives 
would  be  fully  justified  in  terms  of  strengthened  intelligenee  performanee  and  worthwhile 
seientifie  diseovery. 


Resume 


Le  present  doeument  resume  deux  jours  de  discussion  entre  des  specialistes  des  sciences 
eognitives/comportementales  et  des  professionnels  de  1’ analyse  du  renseignement  de 
12  pays  et  d’une  organisation  multinationale  qui  ont  partieipe,  a  Ottawa,  a  une  table 
ronde  organisee  eonjointement  par  RDDC,  le  Bureau  du  Conseil  prive  du  Canada,  et  le 
departement  d’Etat  des  Etats-Unis,  les  23  et  24  fevrier  2009.  Les  partieipants  a  eette  table 
ronde  experimentale  ont  examine  si  les  scienees  cognitives/eomportementales  peuvent 
eontribuer  a  I’analyse  du  renseignement,  et  de  quelle  fa9on.  Ils  ont  eonelu  qu’il  y  a  de 
nombreuses  intereonneetions  entre  ees  domaines,  et  que  le  temps  et  les  depenses 
eonsaerees  a  une  eollaboration  plus  poussee  entre  les  scientifiques  et  les  professionnels 
du  renseignement  seraient  pleinement  justifiees,  car  ils  pourraient  ameliorer  le  rendement 
des  serviees  de  renseignement  et  mener  a  de  preeieuses  deeouvertes  seientifiques. 
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Executive  summary 


Summary  Record  of  the  GFF  Community  of  Interest  on  the 
Practice  and  Organization  of  Intelligence  Ottawa  Roundtable: 
What  Can  the  Cognitive  and  Behavioural  Sciences  Contribute  to 
Intelligence  Analysis?:  Towards  a  Collaborative  Agenda  for  the 
Future 

Anthony  Campbell;  David  R.  Mandel;  DRDC  Toronto  CR  2010-012; 
Defence  R&D  Canada  -  Toronto;  January  2010. 

Introduction  or  background:  This  report  summarizes  discussions  in  Ottawa  on 
February  23-24,  2009  between  several  eminent  behavioral  and  cognitive  scientists  and 
intelligence  analysts  and  their  managers  from  12  countries  and  a  multilateral 
organization.  The  55  participants  met  in  an  experimental  roundtable  format  under  the 
auspices  of  the  Community  of  Interest  on  the  Practice  and  Organization  of  Intelligence 
(COI  POI)  which  is  a  subgroup  of  the  Global  Futures  Forum  (GFF).  Its  purpose  is  to 
promote  transnational  and  interdisciplinary  collaboration  at  the  unclassified  level  to 
improve  security  through  strengthened  intelligence  analysis.  The  meeting  explored 
whether  and  how  the  behavioral  and  cognitive  sciences  can  contribute  to  improved 
intelligence  analysis.  The  roundtable  was  co-hosted  by  DRDC,  Canada’s  Privy  Council 
Office,  and  the  Bureau  of  Intelligence  Research  of  the  US  Department  of  State. 

Methods:  The  roundtable  involved  the  unprecedented  bringing  together  of  an 
international  group  of  scientists  and  intelligence  practitioners  from  12  countries  and  one 
multilateral  organization  to  encourage  interdisciplinary  exchanges  and  networks. 
Effective  exchanges  and  network  building  were  promoted  through  two  days  of  formal 
and  informal  discussion  in  small  groups  as  well  as  in  full  plenary.  The  roundtable  format 
involved  a  series  of  prepared  presentations  by  the  scientific  participants  and 
corresponding  commentaries  by  intelligence  practitioners  in  which  the  consequences  of 
the  scientific  insights  presented  were  debated.  In  the  corridors  of  the  meeting  special 
efforts  were  also  made  to  build  professional  contacts  within  as  well  as  between  the 
scientific  and  intelligence  groups. 

Findings:  The  roundtable  arrived  at  consensus  on  the  practical  value  of  the  behavioural 
and  cognitive  sciences  to  intelligence  analysis.  Despite  skepticism  from  some  over¬ 
pressed  analysts  with  unavoidably  short-term  daily  time  frames,  the  meeting  concluded 
that  the  analytic  community  needs  to  invest  time  in  longer  term  applied  research,  and 
applied  scientists  in  the  behavioural  and  cognitive  fields  need  analyst  interlocutors  to 
ensure  the  practical  grounding  of  their  research  in  relevant  fields.  This  reciprocity  of 
interests  was  seen  as  the  basis  for  recommending  to  governments  the  desirability  of 
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further  collaborative  efforts  between  the  two  groups  following  up  on  the  Ottawa 
roundtable. 


Significance:  This  was  an  unprecedented  multinational  meeting  between  behavioural  and 
cognitive  scientists  and  intelligence  analytical  organizations  spanning  three  continents. 
Despite  initial  skepticism  on  the  part  of  some  analysts,  the  roundtable  demonstrated 
substantial  common  ground  between  the  two  groups,  promoted  improved  mutual 
understanding  of  respective  interests  and  needs  between  them,  and  paved  the  way  to  more 
robust  collaboration  in  future  than  would  otherwise  have  been  possible. 
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Anthony  Campbell;  David  R.  Mandel;  DRDC  Toronto  CR  2010-012;  R 
&  D  pour  la  defense  Canada  -  Toronto;  Janvier  2010. 

Introduction  on  contexte:  Le  present  rapport  resume  deux  jours  de  diseussion  entre  des 
speeialistes  des  seienees  eognitives/eomportementales  et  des  professionnels  de  [’analyse 
du  renseignement  de  12  pays  et  d’une  organisation  multinationale.  Les  55  partieipants  se 
sont  reunis  dans  le  eadre  d’une  table  ronde  experimentale  sous  les  auspiees  de  la 
Communaute  d’interets  sur  la  pratique  et  [’organisation  du  renseignement  (COI  POI),  qui 
est  un  sous-groupe  du  Global  Futures  Forum  (GFF).  L’objeetif  de  eette  eommunaute  est 
de  promouvoir  la  eollaboration  transnationale  et  interdisciplinaire  au  niveau  non  elassifie, 
afin  d’ameliorer  la  seeurite  graee  a  une  meilleure  analyse  du  renseignement.  Les 
partieipants  a  la  table  ronde  ont  examine  si  les  seienees  eognitives/eomportementales 
peuvent  eontribuer  a  [’analyse  du  renseignement,  et  de  quelle  fa9on.  Cette  table  ronde 
etait  organisee  eonjointement  par  RDDC,  le  Bureau  du  Conseil  prive  du  Canada,  et  le 
Bureau  of  Intelligenee  Research  du  departement  d’Etat  des  Etats-Unis. 

Methodes:  Cette  table  ronde  a  reuni  -  fait  sans  precedent  -  des  scientiliques  et  des 
professionnels  du  renseignement  de  12  pays  et  d’une  organisation  multinationale,  pour 
encourager  les  echanges  interdisciplinaires  et  la  formation  de  reseaux.  Pendant  deux 
jours,  les  participants  ont  discute  de  fa9on  formelle  et  informelle  en  petits  groupes,  et  il  y 
a  eu  egalement  des  seances  plenieres.  Ees  scientifiques  ont  presente  une  serie  d’ exposes 
prepares  a  I’avance,  et  les  professionnels  du  renseignement  ont  formule  des 
commentaires  sur  les  consequences  de  leurs  propositions,  ce  qui  a  suscite  des  debats. 
Dans  les  coulisses,  des  efforts  speciaux  ont  etc  deployes  pour  etablir  des  liens  entre  les 
scientifiques  et  les  professionnels  du  renseignement,  et  a  I’intmeur  des  deux  groupes. 

Resultats:  Ees  participants  a  la  table  ronde  sont  arrives  a  un  consensus  sur  I’utilite  des 
sciences  eognitives/eomportementales  pour  [’analyse  du  renseignement.  Malgre  le 
scepticisme  exprime  par  certains  analystes  confrontes  a  des  echeances  a  court  terme,  les 
participants  ont  conclu  que  [’analyse  doit  investir  du  temps  dans  la  recherche  appliquee  a 
long  terme,  et  que  les  speeialistes  des  sciences  eognitives/eomportementales  ont  besoin 
d’interlocuteurs  parmi  les  analystes  pour  enraciner  dans  la  pratique  leurs  travaux  de 
recherche.  C’est  sur  eette  reciprocite  d’interets  que  les  participants  se  sont  fondes  pour 
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dire  aux  gouvernements  que  de  nouveaux  efforts  de  eollaboration  sont  souhaitables  entre 
les  deux  groupes,  afin  de  donner  suite  a  la  table  ronde  d’ Ottawa. 

Importance:  Ottawa  a  ete  le  theatre  d’une  reneontre  Internationale  sans  preeedent  entre 
des  specialistes  des  sciences  cognitives/comportementales  et  des  professionnels  de 
I’analyse  du  renseignement  provenant  de  trois  continents.  Malgre  le  scepticisme  initial  de 
certains  analystes,  la  table  ronde  a  degage  un  terrain  d’ entente  substantiel  entre  les  deux 
groupes,  elle  a  favorise  une  plus  grande  comprehension  mutuelle  de  leurs  interets  et  de 
leurs  besoins  respectifs,  et  elle  a  ouvert  la  voie  a  une  collaboration  accrue  dans  les  annees 
a  venir. 
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1  Overview 


This  report  summarizes  diseussions  in  Ottawa  on  February  23-24,  2009,  between  several 
eminent  behavioural  and  cognitive  scientists  and  intelligence  analysts,  educators,  and 
managers  from  12  countries  and  a  multilateral  organization.  The  55  participants  met  in  an 
experimental  roundtable  format  under  the  auspices  of  the  Community  of  Interest  on  the 
Practice  and  Organization  of  Intelligence  (COI  POI)  which  is  a  subgroup  of  the  Global 
Futures  Forum  (GFF).  Its  purpose  is  to  promote  transnational  and  interdisciplinary 
collaboration  at  the  unclassified  level  to  improve  security  through  strengthened 
intelligence  analysis.  The  meeting  explored  whether  and  how  the  behavioural  and 
cognitive  sciences  can  contribute  to  improved  intelligence  analysis.  The  roundtable  was 
co-hosted  by  Defence  Research  and  Development  Canada  (DRDC),  the  International 
Assessment  Staff  (IAS)  of  Canada’s  Privy  Council  Office,  and  the  Bureau  of  Intelligence 
Research  (INR)  of  the  US  Department  of  State. 

The  Ottawa  Roundtable,  as  it  was  called,  involved  the  unprecedented  bringing  together  of 
an  international  group  of  scientists  and  intelligence  practitioners  from  12  countries  and 
one  multilateral  organization  to  encourage  interdisciplinary  exchanges  and  networks. 
Effective  exchanges  and  network  building  were  promoted  through  two  days  of  formal 
and  informal  discussion  in  small  groups  as  well  as  in  full  plenary.  The  roundtable  format 
involved  a  series  of  prepared  presentations  by  the  scientific  participants  and 
corresponding  commentaries  by  intelligence  practitioners  in  which  the  consequences  of 
the  scientific  insights  presented  were  debated.  In  the  corridors  of  the  meeting  special 
efforts  were  also  made  to  build  professional  contacts  within  as  well  as  between  the 
scientific  and  intelligence  groups. 

The  Ottawa  Roundtable  arrived  at  consensus  on  the  practical  value  of  the  behavioural  and 
cognitive  sciences  to  intelligence  analysis.  Despite  skepticism  from  some  over-pressed 
analysts  with  unavoidably  short-term  daily  time  frames,  the  meeting  concluded  that  the 
analytic  community  needs  to  invest  time  in  longer  term  applied  research,  and  applied 
scientists  in  the  behavioural  and  cognitive  fields  need  analyst  interlocutors  to  ensure  the 
practical  grounding  of  their  research  in  relevant  fields.  This  reciprocity  of  interests  was 
seen  as  the  basis  for  recommending  to  governments  the  desirability  of  further 
collaborative  efforts  between  the  two  groups  following  up  on  the  Ottawa  roundtable. 

This  was  an  unprecedented  multinational  meeting  between  behavioural  and  cognitive 
scientists  and  members  from  intelligence  analytical  organizations  spanning  three 
continents.  The  roundtable  demonstrated  substantial  common  ground  between  the  two 
groups,  promoted  improved  mutual  understanding  of  respective  interests  and  needs 
between  them,  and  paved  the  way  to  more  robust  collaboration  in  future  than  would 
otherwise  have  been  possible. 

The  chapters  that  follow  provide  a  record  of  discussion  of  the  Ottawa  Roundtable  and 
follow  the  layout  of  the  meeting’s  agenda.  Where  the  subsections  of  the  present  report 
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describe  a  particular  talk  by  one  or  more  presenters,  the  relevant  subheading  corresponds 
to  the  title  of  the  presentation  that  appeared  in  the  agenda. 
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2  Welcome  from  the  Co-Hosts  and  Introductions 


The  Ottawa  Roundtable  commeneed  at  09:00. 

Tony  Campbell,  Coordinator  of  the  COI  POI  and  moderator  of  the  Ottawa  Roundtable, 
called  the  meeting  to  order  and  welcomed  the  participants,  who  were  subsequently 
welcomed  by  representatives  of  the  three  co-hosts. 

Vincent  Rigby,  Executive  Director  of  the  IAS,  noted  with  pleasure  the  presence  of  55 
participants  from  12  countries  and  one  multilateral  organization.  He  underlined  the 
importance  that  Canada  attaches  to  unclassified  collaboration  in  the  OFF  including  the 
COI  POI.  He  described  the  two  key  organizing  principles  reflected  in  the  Ottawa 
Roundtable  -  i.e.,  communities  of  interest  and  financial  partnership  -  as  significant 
achievements  that  deserve  to  be  further  developed. 

Warren  Fishbein,  OFF  Director  at  INR,  thanked  the  co-hosts  and  congratulated  the 
organizers  of  the  meeting  for  going  beyond  a  single  issue  and  developing  a  bold 
“horizontal”  agenda  on  a  complex  issue  that  needs  to  be  approached  holistically. 
Innovation  in  form  and  topics  is  what  OFF  is  all  about.  He  took  the  opportunity  to  report 
that  the  INR  had  taken  on  the  lead  role  in  Washington  with  regard  to  the  OFF  and  was 
working  with  the  international  Steering  Group  to  plan  for  the  future. 

Robert  Walker,  Assistant  Deputy  Minister  (Science  and  Technology)  in  Canada’s 
Department  of  Defence  and  CEO  of  DRDC,  welcomed  this  meeting  bringing  together  the 
analytical  and  scientific  communities.  It  reflected  both  the  new  convergence  of  defence 
and  national  security  in  scientific  research,  and  the  new  attention  to  the  human  dimension 
after  so  many  years  in  which  defence  science  was  preoccupied  by  machines  and  weapons. 
Issues  such  as  leadership,  shared  trust,  behavioural  effects,  understanding  ourselves  as 
well  as  our  adversaries,  are  some  of  many  examples  of  areas  of  new  interest  for  DRDC. 
These  issues  are  of  a  complexity  that  requires  collaboration  and  dialogue. 

Tony  Campbell  then  reviewed  the  objectives  and  agenda,  noting  the  ongoing  objectives 
of  trust-building,  habits  of  cooperation,  and  the  generation  of  practical  new  knowledge 
through  interaction  and  dialogue.  He  singled  out  Dr.  David  Mandel,  his  fellow  organizer 
of  the  meeting,  for  special  recognition  for  his  hard  work  and  leadership  in  bringing  the 
Roundtable  about.  He  reminded  participants  of  the  “Chatham  House  Rules”  in  which 
only  speakers  could  be  directly  quoted  and  attributed. 
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3  Scientists  Talking  to  Analysts 


The  first  part  of  Day  1  was  largely  devoted  to  scientific  presentations.  These  were  in  turn 
clustered  into  the  following  three  themes: 

a)  The  role  of  science  in  support  of  analytic  integrity 

b)  Representing  and  communicating  information 

c)  Psychological  perspectives  on  emotion,  risk,  and  inferring  intent. 

3.1  The  Role  of  Science  in  Support  of  Analytic  Integrity 

3.1.1  Setting  the  Stage:  The  Role  of  Science  in  Applied  Communities 

David  Mandel  opened  his  presentation  by  noting  that  the  origin  of  DRDC  support  for  the 
COI  POI  was  a  2007  meeting  organized  by  the  US  Office  of  the  Director  of  National 
Intelligence  (ODNI)  that  also  brought  together  behavioural  scientists  and  members  of  the 
intelligence  community.  That  meeting  had  demonstrated  the  value  of  bringing  the  science 
and  technology  and  analytical  communities  together  to  allow  practitioners  in  these  areas 
an  opportunity  to  get  a  deeper  insight  and  “feel”  for  the  other’s  perspective.  However, 
Mandel  noted  that  there  was  insufficient  opportunity  for  the  scientists  and  practitioners  to 
talk  to  one  another  in  order  to  better  coordinate  their  understanding  and  interests.  For  that 
reason,  DRDC  and  COI  POI  decided  to  plan  a  workshop  that  would  once  again  bring  the 
two  communities  together,  but  which  would  aim  to  do  so  in  a  manner  that  afforded  more 
opportunities  for  semi-structured  discussion  and  interaction  between  the  scientists  and 
practitioners  around  the  question:  “What  can  the  cognitive  and  behavioural  sciences 
contribute  to  intelligence  analysis?” 

He  noted  that  there  is  a  substantial  body  of  literature  that  indirectly  addresses  this 
question  including,  for  example,  decades  of  research  on  judgment  and  decision-making. 
Most  of  it,  however,  remains  in  a  form  that  is  of  limited  use  to  the  intelligence 
community,  not  to  mention  outside  its  awareness.  Of  course,  that  literature,  which  is 
mainly  geared  towards  academic  theory  testing,  is  not  written  in  a  manner  that  speaks 
directly  to  intelligence  issues.  Thus,  bringing  eminent  scientists  that  have  a  track  record 
of  concern  with  applied  issues  together  with  practitioners  in  the  intelligence  community 
offers  an  important  opportunity  to  communicate  some  key  results  and  ideas  to  the 
intelligence  community  and  have  them,  in  turn,  respond  to  those  ideas  and  express  their 
own  views  about  how  science  might  help  improve  the  practice  and  organization  of 
analysis.  By  having  analysts  and  other  members  of  the  intelligence  community  directly 
engaged,  scientists  may  be  better  able  to  carve  out  research  directions  for  the  future  that 
would  directly  address  analytic  concerns. 
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Through  their  expertise  in  seientifie  researeh  and  statistical  methods,  cognitive  and 
behavioural  scientists  are  particularly  well  suited  to  assess  the  applicability  of  possible 
analytical  innovations  just  as  the  analytical  community  is  best  able  to  guide  and  ensure 
the  relevance  of  science  and  technology  in  this  area  to  its  practical  and  organizational 
needs.  The  COI  POI  was  seen  as  an  appropriate  setting  for  this  dialogue  and,  for  that 
reason,  the  sub-theme  of  the  Roundtable  is  “towards  a  collaborative  agenda  for  the 
future.” 

3.1.2  The  Objective  Analysis  of  Analysis 

Paul  Lehner  began  his  presentation  with  three  main  conclusions: 

a)  Intuition,  experience  and  expertise  are  an  inadequate  basis  for  determining  how  to 
improve  expert  practices  in  intelligence  analysis  -  accepted  best  practices  are 
often  ineffective,  and  effective  practices  are  often  counterintuitive. 

b)  Science  can  help  -  objective  scientific  analysis  is  needed  to  help  distinguish 
which  analytic  practices  really  do  work  and  which  ones  don’t. 

c)  Effective  best  practices  are  often  counterintuitive  so  expect  to  be  surprised  -  but 
not  deterred  since  collecting  scientific  data  does  not  have  to  be  a  burden. 

He  emphasized  the  importance  of  objective  scientific  analysis  and  warned  against 
practitioners  evaluating  their  own  practices.  He  sought  to  disprove  that  greater  expertise 
leads  to  more  reliable  results,  using  the  example  in  which  scientific  testing  showed  that  in 
deception  (lie)  detection,  experts  were  54%  accurate,  on  average  -  a  value  obviously  very 
close  to  the  50%  accuracy  rate  expected  by  pure  guesswork  or  random  chance.  Lehner 
noted  that  testing  of  horse-race  handicappers  suggests  that  information  saturation  seems 
to  increase  expert  overconfidence,  not  accuracy.  Thus,  more  information  increases  our 
sense  of  certainty  but  not  necessarily  the  quality  of  our  judgments  and  decisions. 

Biases  have  been  shown  scientifically  to  substantially  impact  expert  judgment. 

“Judgment  bias”  involves  drawing  conclusions  with  insufficient  data.  “Confirmation 
bias”  involves  ignoring  or  under-weighting  information  that  is  inconsistent  with  the 
expert’s  current  views.  Over-confidence  involves  overestimating  the  certainty  of  the 
expert’s  judgment  (and  therefore  the  integrity  and  utility  of  the  practices  that  led  to  those 
judgments).  “Hindsight  bias”,  as  Tetlock  found  in  his  landmark  study  of  experts’  political 
judgments,  occurs  when  individuals  believe  after  the  fact  that  they  predicted  outcomes 
different  than  they  actually  had  predicted,  with  a  drift  of  10-15%  in  the  direction  of  what 
had  actually  happened. 

He  outlined  four  practices  that  are  found  to  be  effective: 
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a)  Systematically  average-out  independent  forecasts.  Accurate  forecasts  can  often  be 
achieved  by  disallowing  inter-analyst  discussion;  the  accuracy  of  consensus  forecasts 
is  about  the  same  as  random  forecasts  (namely,  guesswork). 

b)  It  should  be  possible  to  mitigate  over-confidence  by  having  analysts  use  scientifically 
tested  tools,  though  Analysis  of  Competing  Hypotheses  (ACH),  which  is  often 
regarded  as  an  effective  method  for  reducing  overconfidence,  has  received  little 
systematic  research  attention. 

c)  Simple  prediction  models  work  well  (start  with  a  set  of  candidate  variables  and  pare 
down). 

d)  Role  play  and  simulations  in  conflict  situations  have  been  proven  to  work  quite  well 
when  forecasting. 

A  key  message  was  that  expertise  and  experience  are  a  poor  foundation  -  poorer  than 
objective  scientific  research  -  for  determining  how  to  improve  analytical  practices.  But 
there  is  substantial  room  to  improve  them  and  “we  know  how  to  do  it.”  Nevertheless, 
most  “good  pet  ideas”  do  not  work  and  one  should  expect  to  be  shocked  by  the  results. 
“Finding  the  few  good  ideas  that  work  takes  lots  of  investment.”  And  it  also  requires 
overcoming  “methodological  bias”  -  “without  objective  testing,  we  are  far  too  easily 
convinced  that  our  methods  work.” 

3.2  Representing  and  Communicating  information 

3.2.1  Integrating  Formal  and  Narrative  Analysis 

Baruch  Fischhoff  outlined  the  analyst’s  key  challenges:  to  develop  information  relevant 
to  the  users’  needs,  to  communicate  clearly  and  concisely,  to  capture  assumptions,  and  to 
ensure  a  robust  outcome.  However,  in  addition  there  is  a  cognitive  challenge  in  which  the 
analyst  has  to  integrate  two  different  ways  of  knowing;  formal  versus  narrative.  This 
requires  an  integrative  approach  adapted  to  intelligence  applications.  And  it  involves 
implementation  challenges  because  the  problems  are  complex,  non-linear,  etc. 

Behavioural  research  shows  that  “people  are  simple”  but  predicting  decision  making  is 
difficult  because  behavioural  research  shows  that  the  set  of  relevant  cognitive  principles 
is  large,  the  contextual  triggers  are  subtle,  and  the  interactions  are  complex.  He  zeroed  in 
on  six  relevant  cognitive  principles: 

a)  People  rely  on  imperfect  heuristics  and  therefore  we  need  to  see  cognition  in  the 
context  of  task  demands  and  avoid  sweeping  generalizations. 

b)  Self-reflection  is  difficult,  both  concurrently  and  in  hindsight,  and  therefore  we 
need  the  ability  to  calibrate  performance  empirically. 

c)  People  can’t  change  their  ways  of  thinking  quickly  without  losing  cognitive 
control,  therefore  we  need  measured  transitions. 
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d)  Overconfidence  arises  from  unwitting  omissions,  perhaps  novel,  perhaps 
conventional,  and  therefore  we  need  transparent,  theoretically  neutral  platforms. 

e)  People  need  integrative  cognitive  structures  for  complex  inferences  and  therefore 
we  need  to  structure  analysis. 

f)  People  overestimate  how  well  they  read  others’  minds  (when  extracting  and 
communicating  information);  therefore  we  need  two-way  communication  between 
analysts  and  clients. 

He  went  on  to  discuss  “a  continuum  of  analytical  approaches”  in  which  he  focused  on  the 
efficacy  of  scenarios  and  the  importance  of  integrating  models  and  scenarios.  Scenarios 
can  stretch  minds,  bound  certainty  and  complexity,  help  to  develop  strategies  and 
demonstrate  leadership.  However,  they  are  not  generated  systematically  or  updated,  are 
not  clearly  linked  to  action,  are  not  easily  evaluated,  and  are  not  transparent.  Similarly, 
formal  models  are  claimed  to  accommodate  diverse  forms  of  knowledge  and  aggregate 
information  transparently.  However,  they  can  omit  knowledge  that  is  not  readily 
quantified  or  represented  on  the  team.  They  can  be  too  complex  for  effective  review, 
have  exaggerated  precision,  and  leave  users  without  a  clear  shared  picture. 

Therefore,  it  is  necessary  to  integrate  models  and  scenarios  in  order  to  provide  a 
sufficiently  flexible  conceptual  structure  to  users.  This  balances  formal  and  narrative 
insight,  values  completeness  and  conceptual  precision,  and  allows  analysts  to  maintain 
cognitive  control  while  providing  a  platform  for  deliberation  that  allows  us  to  balance 
formal  and  narrative  insights  and  remain  neutral  on  content  and  conclusions.  He 
presented  a  “mental  models”  approach,  which  uses  influence  diagrams,  as  one  viable 
solution  for  integrating  insights  from  scenario-based  assessments  with  more  formal 
approaches.  Mental  models  can  be  specified  with  a  great  deal  of  precision  (e.g.,  by 
quantifying  the  strength  of  relationships  between  elements  of  a  model)  or  left  imprecise 
as  an  overall  representation  of  subject  matter  expertise. 

3.2.2  Communicating  about  Risk  and  Causaiity 

Denis  Hilton  addressed  how  to  optimize  communications  once  the  analysis  is  done  to 
facilitate  decision-making.  He  noted  that  while  training  in  thinking  skills  has  had  limited 
success  in  improving  decision-making,  simple  changes  in  the  way  information  is 
communicated  can  provide  substantial  improvements  in  decision-making.  In  particular, 
we  can  improve  communications  about  probability  and  causality  through  attention  to: 

a)  conversational  norms  that  govern  human  communication  and  guide  attention  to 
what  is  relevant,  and 

b)  representational  formats  that  improve  the  ability  of  analysts  to  piece  together 
information  in  ways  that  ought  to  improve  the  integrity  (e.g.,  accuracy  and 
coherence)  of  analysis. 
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The  aim  of  his  talk  was  to  encourage  the  design  of  efficient  communications  that  follow 
principles  of  relevance,  avoid  redundancy,  and  are  aware  of  framing  effects  and  order 
effects.  In  addition,  he  aimed  to  offer  a  guide  for  the  choice  of  good  “external 
representations”  with  the  goal  of  capitalizing  on  people’s  “visio-spatial  abilities” 
whenever  possible. 

To  begin  with,  he  outlined  some  general  rules  of  communication  in  organizations, 
distinguishing  between  verbal  and  written  communication.  He  underlined  the  importance 
of  recognizing  the  two  perspectives  in  communication  design:  the  speaker’s  and  the 
hearer’s  or  the  encoder’s  and  decoder’s.  He  noted  that,  in  experiments  designed  to  test 
hypotheses  about  thinking,  the  scientists  had  noted  certain  “communicational  effects”  on 
judgments  and  decisions.  One  implication  of  this  is  that  how  a  report  is  written  will  lead 
people  to  or  away  from  interpretational  errors. 

His  second  broad  theme  addressed  the  decisions  that  need  to  be  taken  in  communication 
-  decisions  that  bear  on  what,  when  and  how  to  transmit  messages.  In  preparing  a  report, 
there  are  three  key  decisions:  what  information  to  include  or  exclude,  the  order  of 
presentation  of  the  information,  and  the  framing  of  that  information. 

In  selecting  or  omitting  information,  there  is  a  dilution  effect.  If  an  item  is  included,  the 
receiver  expects  that  it  will  be  relevant.  However,  in  an  information  “tsunami”,  the  vital 
information  can  be  lost  or  “diluted”  by  other  less  vital  information.  The  implication,  then, 
is  that  more  information  is  not  necessarily  better  for  judgment  and  decision-making. 
Moreover,  conversational  norms  play  a  role  in  this  effect  people  feel  they  should  use  all 
the  information  presented;  namely,  that  if  it  was  communicated,  then  it  must  have  had 
relevance. 

The  ordering  of  information  is  another  key  decision  in  communication  because  studies 
show  that  it  has  a  robust  effect  on  impression  formation  and  other  judgments.  With 
“primacy  effects”  information  inserted  earlier  tends  to  receive  most  attention  in 
judgment,  whereas  with  “recency  effects”  later  information  received  is  given  more 
weight.  Other  factors,  such  as  the  delay  between  information  acquisition  and  judgment,  in 
turn,  moderate  whether  primacy  or  recency  effects  would  be  more  likely  to  occur  in  a 
given  analytical  situation.  Hilton  noted,  as  well,  that  the  evaluative  tone  of  the  message 
can  also  subtly  influence  the  way  messages  are  interpreted.  Order  effects  on  judgment  are 
real,  and  tend  to  easily  escape  notice. 

Dr.  Hilton  then  turned  to  decisions  to  be  taken  about  graphical  representations.  A 
substantial  part  of  the  brain,  he  proposed,  is  dedicated  to  visio-spatial  processing  making 
it  desirable  to  use  graphical  representations  to  tap  that  processing  ability.  However, 
decisions  about  representation  must  be  made  carefully  since  poorly  represented  graphical 
information  has  been  shown  to  be  able  to  cloud  judgment,  whereas  good  graphical  design 
can  improve  judgment.  His  advice  was  to  structure  graphical  information  in  ways  that 
conform  to  regular  conversational  rules  or  norms,  including  a  “scaffold”  structure  that 
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combines  conversational  norms  with  what  we  know  about  the  strengths  and  limitations  of 
human  information  proeessing. 

He  eoncluded  with  some  reeommendations.  First,  keep  it  simple  and  smart.  Beware  of 
order  effects  and  framing  effects  and  test  your  planned  approaeh  against  alternative  order 
and  framing  approaehes.  Pay  attention  to  the  “eonversational”  aspeets  of  graphieal 
representations  -  i.e.,  be  relevant,  sueeinet,  and  use  symbols  that  the  reader  knows. 
Finally,  be  aware  of  human  information  proeessing  propensities:  when  proeessing  verbal 
information,  narrative  “stories”  may  take  hold  easily.  This  may  be  good  for  motivating  a 
deeision,  but  it  ean  also  distort  pereeptions  if  the  narrative  “goes  beyond”  the 
eommunieated  facts.  Secondly,  elicit  visio-spatial  processing  through  graphieal 
presentations  when  you  can.  To  draw  on  Pascal,  the  reader’s  eye  sees  things  that  the  mind 
eomprehends  not. 

3.3  Psychological  Perspectives  on  Emotion,  Risk,  and 
Inferring  Intent 

3.3.1  Risk  as  Analysis  and  Risk  as  Feelings:  Some  Thoughts  about 
Affect,  Reason,  Risk,  and  Rationality 

Paul  Slovie  addressed  issues  of  risk  pereeption  relevant  to  analysis  with  emphasis  on 
affeet,  reason,  risk  and  rationality.  He  noted  that  the  language  of  risk  is  problematic 
because  there  is  no  sueh  thing  as  an  objeetive  risk.  Rather,  humans  invented  the  eoneept 
to  help  them  eope  with  dangers.  The  term  has  multiple  meanings  ranging  from  “hazard”, 
to  probability,  to  eonsequence  (severity).  It  is  a  “judgment-laden”  term  often  refleeting 
underlying  values  in  whieh  one  also  has  to  allow  for  pereeption,  whieh  has  a  reality  of  its 
own. 

Risk  is  multidimensional  and  an  exereise  of  power.  Whoever  defines  risk  defines 
priorities.  Aeeeptanee  of  risk  tends  to  be  redueed  if  (1)  the  hazard  is  new,  (2)  exposure  is 
involuntary,  and  (3)  it  involves  human  failure  as  opposed  to  being  natural.  Risk 
perceptions  and  eognitions  interact  with  social  and  institutional  forces  that  ean  trigger 
massive  social  impacts,  including  “stigma  effects”.  Accidents  ean  be  seen  as  signals  that 
provide  valuable  new  information  about  the  likelihood  of  similar  or  more  destruetive 
mishaps.  Signals  ean  have  “allegorieal  import”  in  terms  of  what  the  event  portends.  This 
ean  lead  to  a  soeial  amplifieation  of  risk  (ripple  effects)  where  the  signals  determine  how 
individuals  and  media  pereeive  the  seriousness  of  the  risk.  It  follows  that  risk  pereeption 
researeh  is  useful  for  analysts  of  risk  in  that  it  offers  insights  that  are  predietive  of  publie 
pereeptions  of  risk. 

Humans  typieally  rely  on  their  gut  feelings  to  perceive  and  respond  to  risk  -  “the 
eommon  eurreney”  of  affeet  influenees  risk  pereeption.  It  mainly  arises  out  of  our 
soeialization.  It  is  also  shaped  by  “neuro-eeonomies”  beeause  the  brain  is  an  “eeonomie 
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risk  assessment  maehine”.  Damasio’s  researeh  showed  that  even  when  we  are  reasoning, 
we  are  feeling.  Thus  we  think  in  two  ways:  intuitively  (affeetive)  and  analytieally.  We 
need  both  systems  to  behave  optimally. 

In  this  affeetive  way  of  reasoning,  images  are  espeeially  important  drivers  of  affeet 
(feelings).  These  in  turn  drive  behaviour  ineluding  deeisions.  But  he  warned  analysts  to 
remember  that  images  are  not  just  pietures  -  words  ean  also  ereate  affeetive  images  with 
powerful  effeet  on  the  feelings  and  deeisions  of  the  deeision-maker. 

3.3.2  Thematic  Analysis  of  Leaders’  Statements  as  Indicators  of  Intent 

Peter  Suedfeld  addressed  thematie  eontent  analysis  and  integrative  eomplexity  in  asking 
the  question  -  how  ean  psyehology  help  analysis  to  enter  “the  thought  proeess  of  an 
enemy  eommander  and  prediet  what  eourse  of  aetion  he  intends  to  pursue.” 

Thematie  eontent  analysis  of  arehival  sourees  is  an  important  approaeh.  It  involves 
seoring  the  level  of  speeifie  thematie  variables,  quantitative  statisties  using  reliable 
(tested)  tools,  and  replieable  and  rigorous  methods.  It  is  not  merely  eounting  word  or 
phrase  frequeneies.  It  has  advantages  (e.g.,  it  is  non-intrusive,  flexible)  and  disadvantages 
(e.g.,  it  has  uneertain  internal  validity,  aeeess  to  important  souree  materials  may  be 
restrieted  or  unavailable). 

Integrative  eomplexity  is  a  thematie  eontent-analytie  teehnique  for  analyzing  the 
strueture,  not  eontent,  of  statements.  It  ean  be  seored.  It  evaluates  the  level  at  whieh 
information  is  being  proeessed  in  a  given  situation.  It  involves  differentiation  (the 
pereeption  of  more  that  one  dimension  of  a  stimulus  or  more  than  one  legitimate  attitude 
about  it)  and  integration  (the  eombination  of  differentiated  pereepts  in  mutual  interaetion, 
synthesis,  trade-off,  or  “as  units  within  a  super-ordinate  sehema”).  It  is  useful  in 
analyzing  individual  information  proeessing  and  deeision-making  questions. 

Where  there  is  high  integrative  eomplexity  there  is  a  high-level  information  seareh, 
eonsideration  of  others’  viewpoints,  flexible  planning  and  strategizing,  toleranee  for 
uneertainty  and  ambiguity,  and  aeeeptanee  of  nuaneed  trade-offs.  The  implieations  of  low 
integrative  eomplexity  inelude  a  limited  information  seareh,  rigid  following  of 
established  plans,  blaek-white  (good-bad)  evaluation,  or  rapid  deeisions  favouring 
simple,  all-or-nothing  strategies  and  solutions.  Integrative  eomplexity  ehanges  under 
stress,  and  these  variations  in  integrative  eomplexity  prediet  sueeess  or  failure  of  a 
leader’s  aetions  in  a  variety  of  situations.  People  maintaining  or  inereasing  integrative 
eomplexity  may  be  able  to  sustain  multiple  roles,  parallel  proeessing  and  a  high 
information  load,  but  they  may  be  slow  and  uneertain  in  making  deeisions.  Signifieant 
reduetions  in  integrative  eomplexity  under  stress  indieate  rigid,  uni-dimensional,  self- 
foeused  thinking  and  a  failure  to  proeess  information.  However,  low  integrative 
eomplexity  allows  for  rapid,  deeisive  deeisions. 
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Integrative  complexity  can  predict  crisis  outcomes  including  situations  where  conflicts 
spiral  -  i.e.,  escalate  or  de-escalate  -  or  it  can  reveal  aggressive  intentions  with  regard  to 
possible  surprise  strategic  attacks  -  or  it  can  help  predict  whether  “simmering  pots”  will 
boil  over.  Thus,  integrative  complexity  analysis  can  be  used  to  identify  behavioral 
patterns  in  crises  that  are  reliable  across  nations  and  historical  periods. 
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4  Analysts  Talking  to  Scientists:  Applications  and 
Implications  for  Intelligence  Analysis 


The  second  major  part  of  the  first  day  of  the  Ottawa  Roundtable  was  devoted  to 
reflections  by  members  of  the  intelligence  community  regarding  the  presentations  they 
had  just  heard.  First,  a  four-person  panel  of  practitioners  was  asked  to  reflect  on  the 
applications  and  implications  for  intelligence  analysis  of  what  they  had  heard  from  the 
initial  speakers.  Following  the  panel  discussion,  small  working  groups,  each  comprised  of 
scientists  and  practitioners,  were  formed  with  the  aim  of  having  the  members  explore  a 
range  of  questions  dealing  with  the  behavioural  science  of  intelligence  analysis.  Each  of 
the  groups  reported  back  during  a  subsequent  plenary  session,  which  culminated  in  a 
general  discussion. 

4.1  Analytical  Perspectives  Panel 

Ame  Biering  of  Denmark  found  the  scientists’  presentations  interesting  and  most  helpful. 
Applying  their  insights  to  actual  analytical  practices  can  be  difficult,  as  he  had  learned  in 
dealing  with  the  challenges  of  teaching  proper  analytical  techniques.  Nonetheless,  he  was 
inspired  to  link  ideas  that  had  been  presented  to  his  service’s  training  approaches. 

Hilton’s  views  on  communication  were  of  particular  relevance  to  policies  in  his  service 
and  his  views  tended  to  usefully  confirm  them.  He  was  also  interested  in  Suedfeld’s 
findings  (on  leaders’  statements  as  indicators  of  intent)  and  he  wondered  how  they  might 
be  compared  to  classified  intelligence  warnings.  However,  he  noted  the  risk  that  leaders 
could  possibly  use  those  findings  to  deceive  analysis. 

Carmen  Medina  of  the  US  noted  that  all  the  issues  raised  were  at  the  centre  of  analytical 
interest.  They  touched  not  only  on  the  process  but  also  on  the  substance  of  analysis,  since 
we  are  not  so  much  analysts  of  countries  as  much  as  analysts  of  behaviours.  The 
presentations  underlined  the  importance  of  open  source  material  and  the  concomitant  use 
of  deep  data  techniques.  She  encouraged  managers  to  expose  their  colleagues  to  the  ideas 
presented  by  the  scientists.  In  that  regard  she  commended  a  website  (www.tedtalks.com) 
for  a  talk  on  cognition  and  the  different  parts  of  the  human  brain  which  she  had  found 
helpful  (see  www.voutube.com/watch?v=UvviU8fzEYU). 

Glenn  Robinson  of  Canada  commented  that  intelligence  managers  and  analysts  are 
concerned  about  how  scientific  insights  can  be  applied  to  their  work  where  the  urgent  so 
often  has  to  trump  the  desirable.  Analysts  constantly  have  too  much  data  and  too  little 
time.  It  is  a  challenge  for  managers  to  deal  with  this  situation  and  foster  improved  results. 
This  raises  the  question  of  whether  techniques  are  more  important  than  knowledge.  With 
those  practical  caveats,  he  went  on  to  identify  the  “take  away”  points  he  had  found  most 
valuable;  the  importance  of  challenging  biases,  the  difficulty  of  overcoming  standard 
approaches,  the  importance  of  effective  presentation,  the  idea  of  combining  narrative  and 
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modeling  to  validate  analysis  and  to  give  these  to  a  elient  to  demonstrate  that  an 
assessment  has  a  system  behind  it  and  is  not  just  impressionistie,  and  finally,  the 
importanee  of  language  in  the  shaping  of  a  report.  He  had  eome  to  the  Roundtable  as  a 
skeptie  and  was  leaving  realizing  that  analysts  will  benefit  from  openness  to  what  seienee 
ean  offer.  The  ehallenge  is  how  to  bring  the  seientists  into  eollaboration  when  their 
eontribution  will  be  seen  as  having  a  “long  term  paybaek”  in  a  short  term  eontext.  Even 
so,  he  eould  see  some  praetieal  ehallenges  he  faees  as  a  manager  that  a  seientifie 
perspeetive  eould  help  him  resolve  sueh  as  how  to  most  effeetively  eover  a  large  area 
with  few  analysts.  How  eould  one  organize  sueh  a  eontribution  without  losing  mueh 
time? 

Elaine  Ruffell  of  the  ElK  found  that  the  presentations  demonstrated  “the  more  you  know, 
the  less  you  aetually  know”.  By  referenee  to  the  Myers-Briggs  personality  profiling 
method  with  whieh  she  was  familiar,  she  deseribed  the  analytieal  eommunity  as 
introverted  and  with  a  elear  preferenee  for  dealing  with  the  tangible,  no  trust  of  the  “6**' 
Sense”,  task-foeused  and  disliking  of  ambiguity.  Clients  tend  to  be  Introverted  and 
Judging  whieh  makes  matters  worse.  Eor  that  reason  she  thought  there  should  be  more 
extroverted  and  intuitive  types  hired  for  analysis  with  openness  to  Eeeling  and 
Pereeiving.  They  are  people-driven,  love  ambiguity  and  are  the  nightmare  of  analysts. 
These  observations  indieate  the  diffieulty  she  will  faee  in  speaking  at  home  about  the 
ideas  that  had  been  put  forward  -  it  will  be  seen  as  a  different  universe.  How  does  one 
bridge  the  gap? 

4.2  Plenary  Reports  from  Small  Groups  and  General 
Discussion 

Several  important  issues  or  themes  emerged  from  the  small  group  diseussions.  One, 
direetly  linked  to  the  overarehing  topie  of  the  Roundtable,  was  that  more  effort  and  time 
needs  to  be  dedieated  to  eonvey  to  intelligenee  analysts  the  relevanee  of  seientifie 
researeh  and  theorizing  to  analytieal  work.  There  was  broad  eonsensus  on  the  importanee 
of  aehieving  that  goal.  Most  analysts  seemed  to  believe  they  are  going  about  their  work 
“soientifieally,”  yet  there  was  reeognition  after  the  day’s  presentations  that  aetual 
analytieal  aetivity  has  little  basis  in  seienee  and  seientists  would  not  eonsider  analysts  as 
really  using  seientifie  methods.  Aeeordingly,  there  was  appreeiation  of  the  value  of 
“outside-in”  thinking  sueh  as  listening  to  behavioural  seientists  and  familiarizing 
themselves  with  their  “open-souree”  findings.  At  the  same  time,  however,  some  groups 
noted  that  translating  the  insights  of  seientists  into  praetieal  applieations  for  analysts 
remains  a  eentral  ehallenge.  Reeognition  of  the  need  to  foeus  more  on  understanding  the 
elient  better  from  a  behavioural  seienee  point  of  view  was  seen  as  a  signifieant  benefit  of 
the  day’s  diseussion. 

There  were  a  number  of  areas  where  the  groups  saw  opportunities  for  building  bridges 
between  seienee  and  praetiee.  Eor  instanee,  many  agreed  that  the  value  of  various  forms 
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of  content  analysis  should  be  further  studied  and  tested.  As  well,  it  was  acknowledged 
that  dealing  with  probability  is  important.  Some  groups  discussed  whether  analysts’ 
confidence  levels  should  be  described  in  terms  of  numeric  odds  or  percentages,  and 
which  approach  better  represents  good  science.  The  conclusion  was  that  researchers 
could  help  to  determine  whieh  was  most  effeetive  for  which  kind  of  problem.  The 
presentations  concerning  the  use  of  language  were  reeognized  as  being  of  fundamental 
importanee  to  analysts.  Vague  predictive  words  should  be  shunned  (e.g.,  “maybe”, 
“could”),  and  behavioural  science  research  could  shed  light  on  the  vagueness  and 
ambiguity  of  different  terms. 

In  a  related  vein,  there  was  debate  about  the  degree  to  whieh  behaviour  ean  be  predicted 
accurately.  There  was  some  debate  about  the  importance  of  achieving  predictive 
aecuracy.  Some  participants  advocated  for  the  importance  of  striving  for  and  measuring 
predictive  aceuraey.  Others,  mainly  in  the  analytie  community,  saw  their  “value  added” 
in  the  assessment  and  understanding  of  situations  with  less  concern  for  numerical 
prediction.  Sometimes,  providing  the  coneeptual  framework  and  identifying  key  drivers 
is  just  as  valuable  to  elients  as  “the  forecast.” 

The  importance  of  developing  visualization  teehniques  was  also  highlighted.  There  was 
concurrence  that  greater  prominence  to  models  and  pictures  is  desirable  but  it  was  also 
aeknowledged  that  there  are  individual  differenees  in  preferences  for  words  versus 
pictures.  Some  groups  suggested  that  elients  prefer  having  some  graphies  over  purely 
narrative  reports  and  that,  for  that  reason  alone,  visualization  in  analysis  is  bound  to 
increase.  If  there  is  greater  use  of  visualization,  there  is  a  ehallenge  to  capturing 
complexity  in  graphics.  There  ean  be  distortion  through  simplilioation.  And  nuanees  can 
be  lost  and  biases  subtly  introduced.  Again  this  was  seen  as  an  area  where  behavioural 
scienee  could  help  shed  light. 

Some  noted  that  there  was  little  mention  in  the  discussion  concerning  matching  analytical 
tools  to  analytical  techniques.  There  was  also  skepticism  voiced  by  some  in  the  analytie 
community  of  the  contention  that  collective  approaehes  have  limited  and  possibly 
counterproductive  value. 

There  was  discussion  of  how  the  issues  raised  at  the  Roundtable  might  bear  on  a  new 
generation  of  analysts.  For  example,  how  does  a  22  year  old  analyst  have  eredibility  with 
a  colonel?  One  suggestion  was  to  expose  the  eolonel  to  the  milieu  and  methods  of  the 
analytical  community.  It  was  accepted  that  credibility  should  be  an  ongoing  concern  and 
seienee  can  be  of  help  in  addressing  this. 

A  follow-up  workplan  might  include  (1)  ongoing  debate  in  the  analytieal  community 
about  the  value  of  computational  prediction  on  the  one  hand  and  eonveying  situational 
and  deseriptive  awareness  on  the  other;  (2)  further  work  to  find  ways  to  translate  the 
ideas  and  coneepts  heard  from  seientists  into  language  analysts  ean  understand,  relate  to 
and  apply  in  practiee;  and  (3)  the  analytieal  community  needs  to  focus  more  on 
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understanding  the  nature  of  the  elient/deeision-maker,  and  here  also  seientists  ean  be  of 
help. 
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5  Dinner  Lecture:  What  Can  Jazz  Contribute  to 
Inteiiigence  Anaiysis? 


Arthur  Katona,  a  retired  Canadian  senior  analytieal  manager,  baeked  up  by  Dr  Jazz,  a 
musieal  eombo,  posed  the  question:  “What  can  jazz  contribute  to  intelligence  analysis?” 

He  pointed  out  that  his  career  had  been  one  in  which  he  did  analysis  by  day  and  played 
music  by  night  and,  despite  obvious  differences,  had  found  a  high  level  of 
complementarity  between  the  two  activities. 

In  his  later  years  most  of  the  most  important  intelligence  assessment  work  was  done 
collectively  in  “peer  groups”  from  other  organizations  even  though  analysts  continued  to 
put  in  a  lot  of  time  alone  and  in  their  own  agencies  maintaining  their  expertise  and 
situational  awareness.  “When  the  groups  met,  the  tone  and  direction  of  discussions,  and 
the  results  were  often  difficult  to  predict  because  something  intangible  would  come  out  of 
the  interaction  of  such  expertise.  The  collective  insights  would  shift,  and  most  people 
around  the  table  would  see  things  in  a  slightly  new  light  and  with  fresh  understanding  - 
when  things  worked  well!” 

Jazz  musicians  like  analysts  spend  a  lot  of  time  alone,  practicing  for  hours  on  end, 
studying,  writing  music,  listening  to  recordings,  and  so  on.  “When  jazz  musicians  get 
together  in  a  band,  in  their  own  version  of  a  peer  group,  they  generally  have  common 
backgrounds,  interests  and  reference  points.”  Yet  when  they  play  together  and  improvise 
collectively,  “musical  conversations  and  listening  are  always  taking  place  and,  in  a  very 
real  sense,  a  constant  series  of  negotiations  is  going  on  among  them  -  like  analysts.” 
Something  intangible  happens  to  everybody  on  the  bandstand.  The  result  is  often  new 
insights,  new  energy,  and  new  interpretations,  just  like  analysts  -  when  things  work 
well!” 

But  of  course,  things  don’t  always  turn  out  as  they  should.  Musicians  are  individuals  with 
many  levels  of  skill,  understanding,  knowledge,  and  concepts  of  self-expression.  There 
are  many  different  and  opposing  schools  of  jazz,  each  with  its  own  proponents  and 
ideologies  -  just  like  intelligence  analysts. 

Here  are  ten  lessons  based  on  the  similarities  between  intelligence  assessment  meetings 
and  jazz  sessions. 

Analyst  meetings  and  jazz  sessions  involve  ongoing  conversations  and 
spontaneous  negotiations  among  participants  so  that  for  success  there  needs  to  be 
agreement  on  common  ground  rules  and  parameters,  either  implicitly  or 
explicitly. 
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Both  analysts  and  musicians  have  to  communicate  effectively  with  an  audience, 
which  includes  getting  feedback  from  that  audience.  Both  have  to  be  concerned 
with  pleasing  the  clients  while,  paradoxically,  realizing  that  they  may  lead  clients 
in  sometimes  unexpected,  uncomfortable  or  even  unwanted  directions.  Both  need 
to  educate  clients  on  the  nature  and  value  of  their  product.  (In  both  cases,  clients 
can  be  real  or  virtual,  and  services  can  be  live  or  recorded.) 

Analyst  meetings  and  jazz  sessions  require  a  skillful  blending  of  individual 
creativities  with  group  dynamics  -  and  yes,  creativity  does  play  a  role  in  the 
intelligence  process. 

Both  demand  the  discipline  to  follow  mainly  unwritten  rules  and  norms,  while 
paradoxically,  pushing  the  envelope  and  striving  to  think  (or  play)  “outside  the 
box”. 

Both  involve  lengthy  and  focused  preparations. 

Both  need  to  use  tactical  judgment  in  responding  quickly  to  unforeseen  situations 
-  to  adjust  on  the  fly,  to  improvise. 

Both  have  had  to  make  significant  strategic  adjustments  over  time  -  e.g.,  jazz 
after  the  bebop  revolution  of  the  1940’s  and  intelligence  analysis  following  the 
Cold  War  and  after  9/11. 

For  both  analysts  and  musicians,  a  little  humility  is  helpful.  Mistakes  are  made  by 
individuals  and  groups.  There  is  a  constant  need  to  self-evaluate,  to  question,  to 
learn,  to  change  and  adapt.  A  balance  needs  to  be  found  between  self-confidence 
and  self-criticism  but  analysts  and  jazz  musicians  need  to  be  able  also  to  maintain 
self-esteem  in  the  face  of  outside  criticism. 

Both  worlds  have  challenges  in  developing  new  generations  of  participants 
including  education,  mentoring,  and  developing  analytical  and  (musical)  skills. 
Finally,  both  involve  a  forward-looking  optic,  while  being  based  on  what  has 
gone  before.  Jazz  musicians  in  their  collective  improvisations  are  simultaneously 
concerned  by  the  present  and  what  is  coming  next  in  the  song  they  are  playing  - 
just  as  how  intelligence  assessments  seek  to  provide  an  understanding  of  how  and 
why  certain  future  outcomes  are  probable. 
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6  Organizational  Perspectives 


The  second  day  of  the  Ottawa  Roundtable  began  with  a  brief  discussion  led  by  Tony 
Campbell  of  thoughts  and  ideas  that  occurred  to  participants  overnight.  The  principal 
thrust  of  comments  was  that  there  are  clearly  all  sorts  of  “crossovers”  between 
intelligence  analysis  and  behavioural  science.  Intelligence  is  fundamentally  about  human 
behaviour  and  the  science  of  human  behaviour  is  obviously  of  high  and  possibly  crucial 
relevance.  These  overnight  reflections  led  into  the  first  main  session  of  Day  2  which 
examined  Organizational  Perspectives. 

6.1  What  Makes  a  Great  Analytic  Team? 

Richard  Hackman  discussed  his  findings  in  scientifically  observing  the  performance  of 
successful  and  unsuccessful  analytic  teams.  This  research  was  challenging  because  the 
target  of  his  studies  -  the  team  and  how  it  functions  -  is  changing,  primarily  because  of 
new  technologies  and  the  accelerating  pace  of  change. 

He  posed  a  puzzle:  why  do  analytic  teams  have  more  resources,  more  flexibility,  and 
more  potential  for  synergy  but  their  performance  is  still  often  “sub-par”?  This  has  often 
been  attributed  to  poor  leadership,  or  to  a  team  being  used  for  work  that  is  better  done  by 
an  individual,  or  the  task  is  too  hard,  or  the  team  has  been  set  up  in  a  way  that  constrains 
its  performance. 

He  described  the  “domain  of  (analytic)  teams”  as  “work  being  performed  by  a  bounded 
group  of  professionals  who  share  responsibility  for  using  data  that  are  incomplete  and/or 
of  uncertain  reliability  to  generate  conclusions  about  something  that  has  happened,  is 
happening,  or  is  likely  to  happen.”  When  forming  a  team  most  of  us  assume  the  key 
element  is  finding  the  best  expertise  but  research  has  shown  that  much  more  is  involved. 

The  first  question  to  address  is  “when  do  we  actually  need  a  team  for  analytic  work?” 
And,  if  so,  “of  what  kind?”  This  requires  us  to  recognize  that  there  are  four  types  of 
teams  depending  on  their  degree  of  “responsibility  for  outcomes”  and  their  level  of 
“synchronicity,”  as  well  as  on  the  distinction  between  the  role  of  individual  members  and 
the  team  as  a  whole.  There  are  “co-acting  groups”  (asynchronous  members,  low 
accountability  for  outcomes),  and  “surgical  teams”  (real  time  interaction),  and  there  are 
“distributed  teams”  (asynchronous  interaction,  high  accountability),  and  face-to-face 
groups  (real  time  interaction,  low  accountability). 

What  are  the  signs  of  an  effective  team?  After  a  study  of  64  units  in  6  different  American 
agencies,  Hackman  and  a  colleague  arrived  at  a  set  of  “highly  reliable”  criteria  for  team 
effectiveness: 
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task  output  is  (at  least)  acceptable  to  those  who  receive,  review,  or  use  it  and  that 
is  occasionally  “magical” 

members  become  increasingly  competent  in  working  together  as  a  team 
members’  personal  growth  and  well-being  are  fostered  by  their  team  experiences 

They  found  that  when  combined  these  three  things  indicate  effectiveness.  The  problem 
for  researchers  was  how  to  measure  these  variables  in  real  terms.  Associated  with  them 
are  process  indicators  of  good  teamwork:  ample  effort,  a  task-appropriate  performance 
strategy,  and  a  full  use  of  member  knowledge  and  skill. 

The  five  conditions  that  strongly  predict  effectiveness  are: 

it  is  a  real  team  (bounded,  stable) 

it  has  a  compelling  direction 

it  has  a  sound  team  structure 

it  has  a  supportive  organizational  context 

it  has  expert  coaching  available 

In  addressing  the  questions,  “what  holds  analytic  teams  back?”  and  “how  can  they  be 
released  to  do  their  work  well?,”  Hackman  identified  the  following: 

A  key  concern  is  failure  to  “bound”  the  team  (i.e.  fluidity  is  good  but  members 
must  know  who  is  part  of  the  team). 

Generally  speaking,  it  is  a  misconception  that  teams  become  less  efficient  over 
time. 

Teams  must  know  their  purpose.  The  best  is  where  a  team  is  given  well-specified 
objectives  or  goals  but  is  allowed  by  their  leader  to  develop  their  own  means  for 
achieving  them.  The  worst  case  is  a  team  that  has  plenty  of  means  and  no  clear 
end  state  envisioned  (as  signalled  by  their  goals).  Teams  with  both  ends  and 
means  also  can  be  a  problem  because  they  usually  waste  resources  (or  fail  to  see 
opportunity  costs;  i.e.,  how  the  same  resources  could  be  applied  toward  achieving 
other  objectives). 

Other  “tripwires”  to  avoid  stumbling  over  include  having  half  a  task  or  less,  team- 
unfriendly  environments,  and  inadequate  team  coaching.  A  major  problem  in  the 
intelligence  community  is  a  tendency  to  divide  a  problem  into  its  components 
instead  of  approaching  the  task  as  a  team  project  (which  would  allow  organizing 
to  balance  one  person’s  weaknesses  and  another’s  strengths). 

Joint  work  is  almost  always  more  productive  if  the  right  conditions  are  met. 
Education  is  a  good  predictor  of  excellent  teams  but  rewards  are  not. 

Coaching  can  play  a  useful  role  especially  in  teams  with  lots  of  expertise. 
“Unguided  groups  with  a  lot  of  expertise  tended  to  do  the  worst”  because  the 
expertise  got  in  the  way.  So  coaching  can  help,  but  only  if  the  team  has  been  well 
designed. 
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The  best  eoaching  is  peer  coaehing.  “Leader  eoaching  usually  helped  a  little,  but 
peer  coaching  predicted  team  effectiveness  better  than  any  other  variable.” 
Research  shows  that  competitive  teams  can  be  counterproductive  because  they 
generate  unhealthy  dynamics  such  as  an  “us  versus  them”  approach,  positioning 
to  win,  strong  pressures  for  conformity,  and  distorted  perceptions,  judgment,  and 
reasoning.  Studies  of  Team  A/Team  B  and  “Devil’s  Advocacy”  approaches  have 
documented  these  problems. 

A  key  “take-away”  from  the  presentation  is  the  need  for  leaders  to  create  or  “enable”  the 
necessary  conditions  for  teams  to  succeed; 

the  cognitive  abilities  of  the  team  should  fit  with  the  demands  of  the  analytic  task 
the  team  should  aggressively  explore  strategies  for  integrating  each  other’s  work 
the  leader  should  coach  at  the  margins 
-  peer  coaching  is  best 

For  further  information  he  referred  to  his  book  “Leading  Teams”  and  a  new  book  “Senior 
Leadership  Teams”.  Or  visit  the  Flarvard  website:  http://www.leadingteams.org.  A  Team 
Diagnostic  Survey  can  be  found  at  https://research.wih.harvard.edu/TDS. 

6.2  The  Intersection:  How  We  are  Organized  and  How  We 
Think 


Josh  Kerbel  focused  on  the  nexus  of  cognitive  science  and  intelligence  analysis, 
particularly  the  interplay  of  language,  thinking,  and  the  metaphors  we  use.  He  used  to 
analyze  Chinese  thinking  but  over  time  he  became  more  interested  in  how  we  think  about 
the  Chinese  than  about  what  they  think. 

The  various  ways  the  US  Intelligence  Community  is  organized  affects  how  we  think 
about  issues;  in  terms  of  organizational  needs,  by  domain  or  discipline,  or  by  intelligence 
type  (imagery,  signals,  human).  Perhaps  of  even  greater  influence  is  the  tendency  to 
assign  each  analyst  their  own  account  which  results  in  a  narrow  account  structure  and 
atomization  of  analysis. 

This  happens  because  we  are  products  of  our  western  intellectual  tradition.  The  standard 
ways  of  thinking  in  analysis  are  based  on  a  reductionist  mindset  where  the  mind  is  like  a 
machine.  Analysts  are  in  the  business  of  predictions  through  trend  analysis, 
disaggregating  the  world  and  encouraging  specialization.  This  approach  reflects  the 
influence  of  Aristotle  and  Newton.  Their  linear  approach  combined  at  the  national 
security  level  with  the  formative  influences  of  the  Cold  War  and  Clausewitz  to  produce 
the  mindset  of  the  intelligence  community.  Specialization  goes  hand  in  hand  with  little 
experience  in  or  incentive  to  do  multi-disciplinary  analysis. 
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If  this  is  the  way  we  think,  is  it  a  good  thing?  There  are  some  advantages  to  the  linear, 
reductionist  model  including  its  simplification  and  predictability.  But  there  are  downsides 
too:  the  world  is  not  a  machine,  it  is  organic,  it  is  increasingly  complex,  and  it  is 
interdependent.  So  our  thinking  model  diverges  from  the  reality  of  the  world  we  live  in.  If 
we  do  not  adapt  to  a  more  syncretic  way  of  thinking,  we  will  be  increasingly  surprised  by 
world  events  and  trends.  But  at  the  same  time,  a  non-linear  thinking  approach  has  a  great 
deal  of  uncertainty. 

So  how  should  we  think?  We  need  to  cultivate  a  different  way  of  thinking  that 
emphasizes  a  world  that  is  fundamentally  organic,  dynamic  and  constantly  changing.  This 
in  turn  requires  that  analysts  are  creative:  if  they  rely  solely  on  critical  thinking  styles, 
they  may  be  caught  heading  in  the  wrong  direction.  Kerbel  warns  against  “throwing  the 
baby  out  with  the  bathwater”  and  suggests  using  complementary,  non-linear  approaches 
that  can  work  alongside  the  more  linear  and  mechanical  approaches.  Some  organizations 
are  pushing  to  adopt  the  former  and  this  fuels  resistance  -  cultural  antibodies  resist  the 
trend. 

These  are  the  situations  where  he  worries  about  critical  thinking.  The  WMD  Commission 
identified  the  need  for  both  more  critical  thinking  and  more  imagination.  The  intelligence 
community  responded  well  to  the  first  task  but  has  done  little  to  address  the  second.  One 
of  the  problems  is  that  it  is  hard  to  train  people  to  be  creative  and  we  tend  not  to  recruit 
for  natural  creativity.  It  is  not  something  that  can  be  covered  in  a  class  or  two.  So  we 
need  to  find  some  “fixes”  to  make  the  analytic  process  more  non-linear  and  more 
dynamic.  These  should  include  encouraging  managers  to  pay  more  attention  to  generating 
multiple  hypotheses,  establishing  analytic  teams,  and  supporting  more  synthesis. 

There  is  a  need  for  smaller,  flatter,  interdisciplinary  teams  where  they  can  go  beyond 
sharing  information  and  actually  create  together.  Team  analysis  has  proven  very  difficult 
to  establish  in  the  bureaucracy.  In  one  organization,  an  initiative  to  create  a  group  that 
would  generate  alternative  hypotheses  and  produce  very  creative  products  foundered 
because  it  focused  on  leaders  when  it  should  have  been  directed  at  analysts.  Analysts  are 
always  generating  hypotheses  but  they  tend  to  give  weight  to  the  one  they  prefer  and  to 
not  give  enough  weight  to  the  alternative  hypotheses. 

He  suggested  that  there  is  a  need  for  a  group  to  address  competing  hypotheses  and  to  get 
analysts  to  use  their  work.  Analysts  talk  about  brainstorming  when  what  they  are  actually 
doing  is  “white  boarding”  which  does  not  involve  any  creativity.  Teams  need  to  be  set  up 
“to  get  it  right,”  even  though  the  track  record  of  teams  has  not  been  encouraging. 
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7  Comparative  Experiences  with  Anaiyst-Scientist 
Cooperation 


7.1  Science  in  Analysis  and  Analysis  in  Science:  A 
Practitioner  and  Private  Sector  Perspective 

Jan  Herring  was  a  former  US  National  Intelligence  Officer  for  Science  and  Technology, 
and  he  subsequently  worked  in  the  private  sector  for  over  20  years,  He  was  involved  in 
technical  intelligence,  which  he  pointed  out  was  not  of  great  utility  on  its  own,  unless  it 
was  melded  with  military,  diplomatic  and  sometimes  business  intelligence.  He  was  asked 
to  pursue  the  creation  of  a  science  and  technology  (S&T)  intelligence  office  to  produce 
S&T  intelligence  for  use  at  various  levels  throughout  the  national  intelligence 
community. 

His  team  hired  S&T  professionals  such  as  engineers  and  scientists,  many  with  “real 
world”  experience  in  a  wide  variety  of  specialized  fields  such  as  chemical  warfare,  air 
defence,  nuclear  weapons,  airborne  weapons,  etc.  Intelligence  specialists  then  trained 
them  as  S&T  analysts  at  a  6-week  program  that  stressed  research  and  writing,  the  CIA’s 
three  elements  (subject  matter  expertise/thought/judgment),  all  source  collection  and 
analysis. 

However,  the  S&T  intelligence  discipline  was  somewhat  different.  It  had  three 
dimensions:  developing  intelligence  product  (assessing  capability,  weapons  and  threats), 
working  on  collection  tools,  and  applying  science  and  technology  to  intelligence 
processes  (such  as  code-breaking,  data  storage  and  retrieval,  etc). 

S&T  intelligence  was  based  on  the  Dewey  Scientific  Methodology.  It  was  focused  on 
“real  world”  and  observable  “factors”  with  the  result  that  it  was  future  oriented  and 
collection  was  closely  linked  to  analysis,  with  analysis  guiding  collection.  This  placed  a 
natural  emphasis  on  all  forms  of  collection:  information  searching  in  every  conceivable 
source  including  repositories  and  databases  of  all  sorts;  technical  sources  including  multi- 
spectral  imaging,  signals  intelligence,  etc;  and  human  intelligence,  which  is  the  ultimate 
source  for  insight  and  intent.  This  in  turn  called  for  collaboration  and  for  a  “team” 
framework. 

These  same  principles  are  true  for  the  business  intelligence  community  where  S&T  input 
to  R&D  planning  and  decisions  is  important  as  it  is  for  sales  and  marketing  (competitive 
positioning),  and  for  mergers  and  acquisitions  and  new  business  development. 

He  then  outlined  some  lessons  learned  about  S&T  intelligence. 

Since  S&T  exists  in  the  real  world, 

o  S&T  intelligence  is  part  of  that  “world” 
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o  scientists  and  engineers  ean  provide  the  best  perspeetive 
o  but  they  must  be  properly  trained  for  intelligenee  work 

Provide  intelligenee  education  and  training; 
o  using  Dewey  Seientifie  Methodology 
o  through  proaetive  eolleetion  and  analysis 
o  by  stressing  all  souree  eolleetion 
o  by  focusing  on  future  capabilities  and  their  impbeations 
o  by  reeognizing  that  a  teehnologieal  foreeast  is  the  best  look  at  the  future 

Provide  a  eareer  development  path: 

o  initial  learning  should  be  under  experieneed  managers, 
o  there  should  be  assignments  to  various  user  organizations  sueh  as 
military/poliey/diplomatic  and  business, 
o  there  should  be  assignments  abroad  and  to  non-governmental 
organizations. 

-  If  S&T  is  a  major  element  in  an  issue,  or  if  it  eould  be  a  determinant  in  its  outeome, 
be  sure  to  inelude  an  S&T  intelligenee  offieer  in  the  analytieal  team. 

7.2  U.S.  Perspectives:  Critical  Thinking  and  Intelligence 
Analysis 


Steven  Rieber  set  out  to  persuade  partieipants  of  three  things: 

(1)  Professional  intelligenee  analysts  eould  be  mueh  better  in  thinking  through 
eomplex  issues. 

(2)  We  need  lots  of  seientifie  researeh  to  find  ways  to  help  people  think  better. 

(3)  Condueting  researeh  is  an  excellent  way  for  intelligenee  and  scienee  to  eooperate. 

Using  a  real  example  of  a  flawed  ehoiee  of  a  dependent  variable,  he  drew  three  lessons: 

(1)  Critieal  thinking  is  essential  -  a  fundamental  mistake  in  the  strueture  of  an 
analysis  ean  undermine  an  entire  analytic  line  of  thought. 

(2)  Critieal  thinking  is  mueh  more  diffieult  than  it  appears  -  even  for  highly 
experieneed  professionals:  intelligent,  well  read  and  refleetive  analysts  were 
undermined  by  their  flawed  argumentation. 

(3)  Learning  the  coneepts  of  eritical  thinking  does  not  ensure  that  one  thinks  well. 
Seleeting  on  the  basis  of  the  dependent  variable  was  understood  by  the  analysts 
but  they  fell  into  the  trap  anyway  -  and  so  did  a  lot  of  others. 

This  is  not  an  isolated  ease.  The  WMD  Commission  reported  numerous  examples  of 
mistakes  in  reasoning.  It  said,  “perhaps  most  troubling,  we  found  an  intelligenee 
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community  in  which  analysts  have  a  difficult  time  stating  their  assumptions  up  front, 
explieitly  explaining  their  logie,  and  in  the  end,  identifying  unambiguously  for 
polieymakers  what  they  do  not  know.” 

This  is  a  big  problem.  The  obvious  solution,  training  analysts  in  critieal  thinking,  has 
some  shorteomings.  The  researeh  using  quite  good  evidenee  shows  that  most  eritieal 
thinking  eourses  are  ineffeetive,  though  some  seem  to  work  well.  But  there  are  gaps  in 
this  research;  the  studies  were  done  on  eollege  students,  not  intelligenee  analysts  or  other 
professionals,  thus  raising  questions  about  how  well  the  extant  findings  and  eonclusions 
drawn  from  them  might  generalize  to  the  intelligenee  domain.  Also,  there  is  almost  no 
researeh  on  how  long  the  gains  in  eritieal  thinking  persist  or  how  well  the  skills  transfer 
from  the  elassroom  to  the  workplaee.  In  addition,  we  do  not  know  how  well  even  the 
most  promising  approaehes  will  work  when  taught  by  instruetors  who  are  less  eommitted 
to  the  approaeh  than  the  people  who  developed  it.  Finally,  nearly  all  the  researeh  on 
eritieal  thinking  and  human  judgment  is  done  at  the  individual  level.  We  know  mueh  less 
about  how  to  help  teams  reason  better. 

All  this  means  that  we  need  to  take  improving  thinking  and  judgment  seriously  as  a 
researeh  area.  It  is  hard  to  imagine  doing  that  but  other  areas  like  medicine  have  shown 
that  radieal  ehanges  based  on  seientific  evidenee  ean  happen.  Why  not  in  intelligenee 
analysis  then? 

He  left  us  with  three  key  points: 

(1)  Sinee  intelligenee  analysis  primarily  involves  thinking  through  eomplex 
issues,  we  need  to  develop  better  ways  to  help  people  think  more  elearly. 

(2)  Most  of  the  researeh  on  improving  thinking  for  the  intelligenee  community 
has  yet  to  be  done. 

(3)  Condueting  this  researeh  together  would  be  a  great  way  for  the  intelligenee 
and  scientifie  eommunities  to  eooperate. 

7.3  Australian  Perspectives:  Simulating  the  Analytic 
Environment  and  Developing  Analyst  Skills 

Colin  Waste  11  and  Piers  Dunean  diseussed  the  SINTELLA  projeet,  whieh  stands  for 
simulation  of  intelligenee  analysis.  Onee  eomplete,  the  goal  is  to  deliver  the  software  tool 
to  analysts  for  training.  More  broadly,  the  researeh  project  seeks  to  identify  heuristies  and 
biases  that  are  present  when  analysts  are  presented  with  a  tasking. 

Drawing  on  earlier  psyehologieal  aeeounts,  they  noted  that  there  is  a  dual-proeess  model 
of  thinking;  the  experiential  system  and  rational  system.  The  experiential  system  is 
automatie,  rapid  and  difficult  to  change.  The  rational  system  is  intentional,  effortful 
proeessing  whieh  is  easier  to  ehange  and  is  heavily  language  based. 
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They  then  outlined  Stage  1  of  SINTELLA  whieh  included  computer  software  with  an  8  x 
8  “information  block”  or  table  offering  64  information  cells.  The  program  requires 
students  to  build  a  travel  advisory  for  fellow  students  by  selecting  information  they  deem 
relevant  from  the  table.  The  software  can  determine  which  cells  were  opened,  the 
duration  they  were  open,  the  words  selected,  and  what  was  copied  into  the  workspace. 
Their  initial  simulation  research  found  that  there  is  a  variation  in  risk  assessment  and  the 
length  of  the  reports  vary. 

They  concluded  by  discussing  future  developments  in  the  project.  They  are  developing  a 
dataset  for  ACH  via  a  crime  scene  scenario  (“murder  who  done  it?”).  This  will  be 
designed  to  eliminate  suspects  but  not  to  be  absolutely  certain  as  to  who  is  the  murderer. 

7.4  Canadian  Perspectives:  A  Caiibration  Study  of  an 
inteiiigence  Assessment  Division 

Alan  Barnes  and  David  Mandel  discussed  an  analyst-scientist  project  that  they  have  been 
working  on  together  studying  analytical  judgments  in  an  assessment  organization.  Alan 
Barnes  began  by  discussing  problems  associated  with  analytical  judgments.  He  noted  the 
problem  of  intelligence  organization  being  too  casual  in  their  approaches  to  judgments 
and  communication.  This  called  for  greater  transparency  on  how  judgments  are  derived, 
how  we  debate  and  discuss  them  with  peers,  and  how  we  communicate  them  to  the  client. 

Initially  Barnes  took  an  informal  approach  in  addressing  both  aspects.  He  thought  one 
way  to  make  judgments  more  transparent  would  be  to  externalize  the  thinking  process 
perhaps  using  ACH  where  appropriate,  or  lists,  belief  probabilities  expressed  as  numbers 
out  of  ten,  and/or  probability  charts. 

To  improve  communication  of  analytical  judgments,  Barnes  and  his  colleagues  searched 
for  precise  terms,  avoided  the  old  vague  terms,  and  covered  a  fully  a  range  of 
probabilities  ending  up  with  a  chart  of  probability  terms.  Over  a  couple  of  years,  a  file 
was  kept  of  raw  material  to  allow  for  a  review  of  judgments.  To  check  for  accuracy  of 
judgments,  they  brought  in  an  outside  analyst  to  carry  out  a  review,  which  involved 
tracking  whether  the  predicted  events  occurred  or  didn’t  over  the  forecasted  period. 

David  Mandel  followed  up  by  outlining  his  current  research  examining  the  quality  of 
predictive  judgments  of  a  strategic  intelligence  division  by  utilizing  the  data  that  Barnes 
and  the  outside  analyst  had  collected.  The  judgment  data  was  taken  from  about  an  18 
month  assessment  period  and  covered  649  predictive  judgments  abstracted  from  5 1 
Intelligence  Memoranda  produced  by  the  division.  Of  the  judgments,  89%  or  580  could 
be  coded  in  terms  of  outcome  (the  remainder  were  ambiguous).  This  was  facilitated  by 
the  practice  in  the  division  of  assigning  a  numeric  probability  estimate  on  an  uncertainty 
scale  (0/10  to  10/10). 
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Mandel  looked  at  two  measures  of  judgment  quality:  calibration  and  discrimination.  An 
analyst  is  well  calibrated  if  his  or  her  probability  judgments  of  an  event  occurrence  match 
actual  relative  frequencies  of  event  occurrence  (over  the  relevant  time  frame). 
Discrimination  can  be  thought  of  as  the  extent  to  which  the  analyst  utilizes  the  full  range 
of  the  uncertainty  scale,  and  can  be  interpreted  as  measure  of  the  proportion  of  explained 
variance  in  the  outcome.  His  findings,  to  date,  indicate  that  the  quality  of  analytic 
judgments  in  the  group  was  “good  to  excellent,”  with  about  90%  of  the  judged  events 
correctly  classified  by  analysts  in  the  division. 

He  noted,  however,  that  the  causal  basis  for  the  high  level  of  performance  observed 
cannot  yet  be  determined,  although  some  factors,  such  as  judgment  difficulty,  issue 
importance,  and  analyst  experience  could  be  ruled  out  since  each  had  a  small  or 
negligible  effect  on  calibration.  Other  plausible  determinants  that  have  yet  to  be 
investigated  include  the  detailed  review  process  in  the  organization  with  multi-expert 
input,  the  availability  of  internal  guidelines  to  promote  analytic  integrity  especially  in  the 
assessment  and  communication  of  uncertainty,  and  environmental  factors,  such  as  ample 
time  and  resources. 

He  concluded  that  the  research  illustrates  the  potential  for  the  intelligence  and  scientific 
communities  to  work  together  in  order  to  validate  the  effectiveness  of  analytic 
performance.  He  also  noted  that  validation  field  research  studies,  such  as  this,  are 
themselves  marks  of  analytic  integrity  because,  in  order  to  improve  capabilities  and 
performance,  one  needs  situational  awareness  of  an  organization’s  current  standing. 
Compared  to  the  measured  results,  intuitions  of  “how  we  are  doing”  and  post-mortem 
analyses  of  intelligence  failures  or  near  misses  are  poor  substitutes.  Thus,  one 
recommendation  was  that  intelligence  organizations  consider  the  value  of  “keeping  track” 
of  their  performance  in  a  systematic  manner  that  allows  for  a  proactive  outcome -based 
accountability  system  to  be  implemented.  Currently,  most  efforts  are  either  process-based 
or  reactive. 
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8  Applications  and  Implications  for  Intelligence 
Analysis 


As  on  Day  1,  during  the  latter  part  of  Day  2,  partieipants  formed  into  small  groups  that 
were  asked  to  discuss  what  they  had  learned  at  the  Roundtable  and  what  should  be  done 
about  it. 

In  their  reports  the  following  issues  were  raised; 

Is  intelligence  analysis  a  guild  or  a  profession?  If  the  former,  it  is  about  tradecraft; 
and,  if  it  is  a  profession,  it  should  be  doing  research  to  build  the  field. 

There  was  discussion  of  synthetic  versus  analytic  versus  integrated  method. 
Analysis  involves  creativity  but  it  involves  breaking  things  into  pieces.  But  there 
eventually  is  a  need  to  cobble  the  bits  together. 

There  was  discussion  of  generational  data  and  it  was  suggested  that  there  should 
be  work  on  what  we  do  right  and  wrong  in  the  development  of  experts. 

One  suggestion  was  that  the  younger,  greener  analysts  should  be  mixed  with  more 
experienced  experts. 

There  is  a  need  to  recognize  and  reward  analysts,  such  as  by  getting  each  service 
to  nominate  their  best  and  give  them  a  chance  to  talk  about  their  views  before 
their  peers. 

Perhaps  COI  POI  should  work  on  identifying  the  criteria  for  the  ideal  analyst. 

The  greatest  improvement  in  the  intelligence  domain  is  coming  from  outside  - 
perhaps  we  should  be  imagining  podcasts  instead  of  papers. 

The  issue  of  partnerships  was  also  raised,  and  three  distinct  applied  groups  were 
identified:  the  analysts,  the  managers,  and  professional  developers. 

There  was  consensus  on  the  importance  of  getting  scientists  to  talk  to  analysts  and 
vice  versa.  Something  better  than  talking  every  two  years  is  required.  There  need 
to  be  studies  to  address  the  challenges  identified  in  the  Roundtable.  Perhaps  we 
should  consider  launching  a  journal  for  the  scientific  side  of  analysis. 

The  importance  of  training  was  reiterated  many  times  and  it  was  recommended 
that  in  training  there  should  be  room  for  a  scientific  dimension  and  contribution  to 
that  training. 

To  advance  in  this  field  of  collaboration,  people  underlined  the  importance  of 
high  level  buy-in.  Intelligence  organizations  are  under  huge  time  pressures,  they 
mainly  work  at  an  intense  tempo,  and  they  have  a  low  tendency  to  look  inward. 

So  the  cycle  has  to  be  broken  to  validate  research  investments. 

There  was  widespread  enthusiasm  by  analysts  for  launching  calibration  studies 
such  as  had  been  described  by  Alan  Barnes  and  David  Mandel  at  the  Roundtable. 
Richard  Hackman’s  ideas,  it  was  noted,  depended  somewhat  on  the  issue  of  scale. 
Since  some  organizations  are  larger  than  others,  perhaps  there  is  a  need  to 
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distinguish  between  “groups”  and  “teams.”  However,  it  was  pointed  out  that  the 
seale  issue  eould  perhaps  be  got  around  by  ereating  virtual  teams. 

It  was  observed  that  partieipants  had  been  more  eonsensual  on  the  seeond  day 
than  on  the  first  day,  in  that  they  reeognized  that  it  is  both  possible  and  neeessary 
to  do  worthwhile  applied  seientifie  researeh  on  analysis,  and  there  is  an  interest  in 
applying  it  in  the  workplaee. 

Insights  from  the  researeh  would  be  interehangeable  among  COI  POI  member 
countries. 

There  was  a  lot  of  discussion  about  training,  including  the  potential  role  of 
behavioural  science  in  empirically  testing  the  effectiveness  of  existing  and 
potential  future  approaches. 

Behavioural  science  research  has  found  that  simple  models  do  very  well  but  today 
we  saw  tension  over  complexity  as  a  fact  of  analytical  life. 

Other  tensions  addressed  included  issues  of  precision,  quantification,  and  the 
“inside  view”  (which  tends  to  rely  on  detail-rich  scenarios  and  imagination  about 
specific  cases)  versus  the  “outside  view”  (which  relies  on  the  use  of  detail-sparse 
statistical  information  aggregated  over  many  cases). 

There  was  debate  over  the  merits  of  aiming  for  accuracy  in  a  context  of  “point 
precision”.  Perhaps  giving  decision  makers  insight  is  more  important  than 
precision  in  predictive  estimates.  On  the  other  hand,  vague  estimates  can  more 
easily  be  misconstrued  and  manipulated,  thus  there  are  good  reasons  for 
improving  at  least  the  precision  of  our  language  of  uncertainty. 
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9  Conclusions  and  Next  Steps 


The  Coordinator  reviewed  the  objectives  of  the  meeting  and  expressed  satisfaction  with 
the  positive  results  obtained  in  each  case.  Networks  of  trust  had  been  developed  with  new 
cross  disciplinary  and  trans-boundary  relationships  cemented.  We  all  learned  practical 
lessons  that  we  can  share  in  our  offices  next  week.  And  we  definitely  created  new 
insights  and  knowledge. 

As  for  next  steps,  he  announced  that  the  next  meeting  of  the  COI  POI  was  scheduled  for 
May  7-8,  2009  in  Bucharest. 

The  co-hosts  drew  the  Ottawa  Roundtable  to  a  conclusion  by  thanking  the  organizers  and 
participants  for  making  it  an  excellent  meeting.  The  Roundtable  had  reached  consensus 
on  the  practical  value  of  the  behavioral  and  cognitive  sciences  to  intelligence  analysis. 
There  is  a  revolution  taking  place  in  which,  as  one  participant  put  it,  “everything  that  is 
simple  is  wrong,  and  everything  that  is  complex  is  useless”.  Analysis  needs  research  and 
applied  scientists  need  analyst  interlocutors  to  help  analysis  and  assessment  to  adapt, 
change  and  improve. 

There  was  also  consensus  that  it  is  important  for  behavioural  scientists  to  translate  the 
ideas  of  the  last  two  days,  which  only  skimmed  the  surface  of  what’s  there,  and  find  a 
way  to  communicate  it  all  to  analysts  in  order  to  pursue  the  goal  of  developing  a 
collaborative  agenda  for  the  future. 

The  meeting  ended  at  3  p.m. 
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List  of  symbols/abbreviations/acronyms/initialisms 


ACH 

Analysis  of  Competing  Hypotheses 

CEO 

Chief  Exeeutive  Officer 

CIA 

Central  Intelligence  Agency 

COI  POI 

Community  of  Interest  on  the  Practice  and  Organization  of  Intelligence 

DND 

Department  of  National  Defence 

DRDC 

Defence  Research  &  Development  Canada 

OFF 

Global  Futures  Forum 

IAS 

International  Assessment  Staff 

INR 

Bureau  of  Intelligence  Research 

ODNI 

Office  of  the  Director  of  National  Intelligence 

R&D 

Research  and  development 

S&T 

Science  and  technology 

SINTEFFA 

Simulation  of  Intelligence  Analysis 

UK 

United  Kingdom 

US 

United  States 

WMD 

Weapons  of  mass  destruction 
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